sync: replace hardcoded concurrency limit with --concurrency flag#5196
sync: replace hardcoded concurrency limit with --concurrency flag#5196luketchang wants to merge 1 commit into
Conversation
The sync engine capped in-flight requests at a hardcoded MaxRequestsInFlight = 20, applied to file PUTs, deletes, mkdirs and rmdirs alike. That ceiling is fine for most workspaces but is too conservative for fast networks / high-tier workspaces (slow uploads of larger app bundles) and too aggressive when the workspace is rate-limiting (we want to dial it down without recompiling). This change: - Adds Concurrency to libs/sync.SyncOptions, normalised to MaxRequestsInFlight in sync.New when unset so existing callers don't need updating. - Threads the limit through groupRunParallel/applyDiff in libs/sync/watchdog.go; the constant is kept as the documented default. - Surfaces a --concurrency int flag on the two user-facing entry points that drive this code path: 'databricks sync' and 'databricks bundle sync' (the latter is what backs the apps deploy/sync flow). - Validates --concurrency >= 1 at flag-parse time per the project rule on rejecting incompatible inputs early. - Adds TestGroupRunParallelHonorsLimit to lock in the parallelism cap. Slack thread: https://replit.slack.com/archives/C0A2Z9042FR/p1778082184841149
Approval status: pending
|
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
The sync engine capped in-flight requests at a hardcoded MaxRequestsInFlight = 20, applied to file PUTs, deletes, mkdirs and rmdirs alike. That ceiling is fine for most workspaces but is too conservative for fast networks / high-tier workspaces (slow uploads of larger app bundles) and too aggressive when the workspace is rate-limiting (we want to dial it down without recompiling).
This change:
Slack thread:
https://replit.slack.com/archives/C0A2Z9042FR/p1778082184841149
Changes
Why
Tests